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WHAT IS CLAIMED: 



1. A method of analyzing performance of an 
interactive voice response (IVR) system of a call processing 
center with respect to routing of calls received by the IVR 
system from callers, the IVR system being operable to route 
the calls in accordance with a call-flow model of the IVR 
system based on inputs of the callers to the IVR system, the 
performance being defined as accuracy of routing decisions 
made by the IVR system compared to where the callers should 
have been routed in accordance with the callers' true reasons 
for calling the call processing center, said method 
comprising the steps of: 

recording from end to end plural calls from the 

callers ; 

for a plurality of recorded calls, coding and 
annotating an agent-caller interaction, including a topic 
discussed during the interaction to obtain coded and 
annotated agent-caller interactions; 

for a plurality of recorded calls, detecting a 
routing operation of the IVR system by detecting a sequence 
of events in the IVR system, with reference to a call-flow 
file that summarizes the call-flow model of the IVR system; 

for a plurality of recorded calls, summarizing the 
detected sequence of events in a call-sequence file; 

compiling a list of call-sequence files; 

compiling from the list of call-sequence files a 
summary file that lists information collected from calls 
interacting with the IVR system, including routing 
information and a routing destination out of the IVR system 
for the calls ; 

creating summary results for the coded and 
annotated agent-caller interactions; and 
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cross- tabulating the routing information in the 
summary file with the summary results from the coded and 
annotated agent-caller interactions . 

2. A method according to claim 1, wherein said 
recording step is performed off-site in relation to the IVR 
system. 



comprising the step of generating, from a result of the 
cross-tabulating step, a confusion matrix that correlates 
actual routing decisions of the IVR system with true reasons 
for calling, as ascertained from annotations of agent-caller 
interactions . 

4. A method according to claim 1, wherein said 
coding and annotating step is performed manually by one or 
more persons listening to the plurality of recorded calls. 

5. A method according to claim 1, wherein said 
coding and annotating step is performed automatically using 
topic-recognition software . 

6. A method according to claim 1, wherein inputs 
to the IVR system comprise speech and/or DTMF tones. 



interactive voice response (IVR) system of a call processing 
center with respect to routing of calls received by the IVR 
system from callers, the IVR system being operable to route 
the calls in accordance with a call-flow model of the IVR 
system based on inputs of callers to the IVR system, the 
performance being defined as accuracy of routing decisions 



3 . 



A method according to claim 1, further 



7 . 



An apparatus for analyzing performance of an 
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made by the IVR system compared to where callers should have 
been routed in accordance with the callers' true reasons for 
calling the call processing center, said apparatus 
comprising : 

means for recording from end to end plural calls 
from callers; 

means for, for a plurality of recorded calls, 
coding and annotating an agent-caller interaction, including 
a topic discussed during the interaction to obtain coded and 
annotated agent-caller interactions; 

means for, for a plurality of recorded calls, 
detecting a routing operation of the IVR system by detecting 
a sequence of events in the IVR system, with reference to a 
call-flow file that summarizes the call-flow model of the IVR 
system; 

means for, for a plurality of recorded calls, 
summarizing the detected sequence of events in a call- 
sequence file; 

means for compiling a list of call-sequence files; 

means for compiling from the list of call-sequence 
files a summary file that lists information collected from 
calls interacting with the IVR system, including routing 
information and a routing destination out of the IVR system 
for the calls ; 

means for creating summary results for the coded 
and annotated agent-caller interactions; and 

means for cross- tabulating the routing information 
in the summary file with the summary results from the coded 
and annotated interactions. 

8. An apparatus according to claim 7, wherein said 
means for recording is located off-site in relation to the 
IVR system. 
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9. An apparatus according to claim 7, further 
comprising means for generating, from cross- tabulated 
information generated by said means for cross-tabulating, a 
confusion matrix that correlates actual routing decisions of 
the IVR system with true reasons for calling, as ascertained 
from annotations of agent-caller interactions. 

10. An apparatus according to claim 7, wherein 
said means for coding and annotating performs coding and 
annotation automatically using topic-recognition software. 



11. An apparatus according to claim 7, wherein 
inputs to the IVR system comprise speech and/or DTMF tones. 

12. A system for analyzing performance of an 
interactive voice response (IVR) system of a call processing 
center with respect to routing of calls received by the IVR 
system from callers, the IVR system being operable to route 
the calls in accordance with a call-flow model of the IVR 
system based on inputs of callers to the IVR system, the 
performance being defined as accuracy of routing decisions 
made by the IVR system compared to where callers should have 
been routed in accordance with the callers' true reasons for 
calling the call processing center, said system being 
operable to: 

record from end to end plural calls from the 

callers ; 

for a plurality of recorded calls, code and 
annotate an agent-caller interaction, including a topic 
discussed during the interaction to obtain coded and 
annotated agent-caller interactions; 
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for a plurality of recorded calls, detect a routing 
operation of the IVR system by detecting a sequence of events 
in the IVR system, with reference to a call-flow file that 
summarizes the call-flow model of the IVR system; 

for a plurality of recorded calls, summarize the 
detected sequence of events in a call-sequence file; 

compile a list of the call-sequence files; 

compile from the list of call-sequence files a 
summary file that lists information collected from callers in 
the IVR system, including routing information and a routing 
destination out of the IVR system for each call; 

create summary results for the coded and annotated 
agent-caller interactions; and 

cross- tabulate the routing information in the 
summary file with the summary results from the coded and 
annotated interactions . 

13. A system according to claim 12, wherein 
recording of calls from end to end is performed off-site in 
relation to the IVR system. 

14. A system according to claim 12, said system 
further being operable to generate, from cross- tabulated 
information, a confusion matrix that correlates actual 
routing decisions of the IVR system with true reasons for 
calling, as ascertained from annotations of agent-caller 
interactions . 

15. A system according to claim 12, wherein coding 
and annotation are performed automatically by a topic- 
recognition algorithm. 
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16. A system according to claim 12, wherein inputs 
to the IVR system comprise speech and/or DTMF tones. 

17 . A method of analyzing performance of an 
interactive voice response (IVR) system of a call processing 
center with respect to routing of calls received by the IVR 
system from callers, the IVR system being operable to route 
the calls in accordance with a call-flow model of the IVR 
system based on inputs of the callers to the IVR system, the 
performance being defined as accuracy of routing decisions 
made by the IVR system compared to where the callers should 
have been routed in accordance with the callers' true reasons 
for calling the call processing center, said method 
comprising the steps of: 

recording from end to end plural calls from the 

callers ; 

for a plurality of recorded calls, coding and 
annotating an agent-caller interaction, including a topic 
discussed during the interaction to obtain coded and 
annotated agent-caller interactions; 

for a plurality of recorded calls, detecting a 
routing operation of the IVR system by detecting a sequence 
of events in the IVR system, with reference to a call-flow 
file that summarizes the call-flow model of the IVR system; 
and 

cross-tabulating routing information obtained from 
the detected sequence of events with results from the coded 
and annotated agent-caller interactions. 



18 . An apparatus for analyzing performance of an 
interactive voice response (IVR) system of a call processing 
center with respect to routing of calls received by the IVR 
system from callers, the IVR system being operable to route 
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the calls in accordance with a call-flow model of the IVR 
system based on inputs of the callers to the IVR system, the 
performance being defined as accuracy of routing decisions 
made by the IVR system compared to where the callers should 
have been routed in accordance with the callers' true reasons 
for calling the call processing center, said apparatus 
comprising : 

means for recording from end to end plural calls 
from the callers; 

means for, for a plurality of recorded calls, 
coding and annotating an agent-caller interaction, including 
a topic discussed during the interaction to obtain coded and 
annotated agent-caller interactions ; 

means for, for a plurality of recorded calls, 
detecting a routing operation of the IVR system by detecting 
a sequence of events in the IVR system, with reference to a 
call-flow file that summarizes the call-flow model of the IVR 
system; and 

means for cross- tabulating routing information 
obtained from the detected sequence of events with results 
from the coded and annotated agent-caller interactions. 

19. A system for analyzing performance of an 
interactive voice response (IVR) system of a call processing 
center with respect to routing of calls received by the IVR 
system from callers, the IVR system being operable to route 
the calls in accordance with a call-flow model of the IVR 
system based on inputs of the callers to the IVR system, the 
performance being defined as accuracy of routing decisions 
made by the IVR system compared to where the callers should 
have been routed in accordance with the callers' true reasons 
for calling the call processing center, said system being 
operable to: 
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record from end to end plural calls from the 

callers ; 

for a plurality of recorded calls, code and 
annotate an agent-caller interaction, including a topic 
discussed during the interaction to obtain coded and 
annotated agent-caller interactions; 

for a plurality of recorded calls, detect a routing 
operation of the IVR system by detecting a sequence of events 
in the IVR system, with reference to a call-flow file that 
summarizes the call-flow model of the IVR system; and 

cross- tabulate routing information obtained from 
the detected sequence of events with results from the coded 
and annotated agent-caller interactions. 

20. A method of analyzing performance of an 
automated response system of a contact processing center with 
respect to routing of contacts received by the automated 
response system from contactors, the automated response 
system being operable to route the contacts in accordance 
with a contact- flow model of the automated response system 
based on inputs of the contactors to the automated response 
system, the performance being defined as accuracy of routing 
decisions made by the automated response system compared to 
where the contactors should have been routed in accordance 
with the contactors' true reasons for contacting the contact 
processing center, said method comprising the steps of: 

recording from end to end plural contacts from the 
contactors ; 

for a plurality of recorded contacts, coding and 
annotating an agent-contactor interaction, including a topic 
discussed during the interaction to obtain coded and 
annotated agent-caller interactions; 
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for a plurality of recorded contacts, detecting a 
routing operation of the automated response system by 
detecting a sequence of events in the automated response 
system, with reference to a contact-flow file that summarizes 
the contact- flow model of the automated response system; and 

cross- tabulating routing information obtained from 
the detected sequence of events with results from the coded 
and annotated agent-contactor interactions. 



21. An apparatus for analyzing performance of an 
automated response system of a contact processing center with 
respect to routing of contacts received by the automated 
response system from contactors, the automated response 
system being operable to route the contacts in accordance 
with a contact-flow model of the automated response system 
based on inputs of the contactors to the automated response 
system, the performance being defined as accuracy of routing 
decisions made by the automated response system compared to 
where the contactors should have been routed in accordance 
with the contactors' true reasons for contacting the contact 
processing center, said apparatus comprising: 

means for recording from end to end plural contacts 
from the contactors; 

means for, for a plurality of recorded contacts, 
coding and annotating an agent-contactor interaction, 
including a topic discussed during the interaction to obtain 
coded and annotated agent-caller interactions; 

means for, for a plurality of recorded contacts, 
detecting a routing operation of the automated response 
system by detecting a sequence of events in the automated 
response system, with reference to a contact-flow file that 
summarizes the contact- flow model of the automated response 
system; and 
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means for cross- tabulating routing information 
obtained from the detected sequence of events with results 
from the coded and annotated agent-contactor interactions. 



22. A system for analyzing performance of an 
automated response system of a contact processing center with 
respect to routing of contacts received by the automated 
response system from contactors, the automated response 
system being operable to route the contacts in accordance 
with a contact- flow model of the automated response system 
based on inputs of the contactors to the automated response 
system, the performance being defined as accuracy of routing 
decisions made by the automated response system compared to 
where the contactors should have been routed in accordance 
with the contactors' true reasons for calling the contact 
processing center, said system being operable to: 

record from end to end plural contacts from the 
contactors ; 

for a plurality of recorded contacts, code and 
annotate an agent-contactor interaction, including a topic 
discussed during the interaction to obtain coded and 
annotated agent-caller interactions; 

for a plurality of recorded contacts, detect a 
routing operation of the automated response system by 
detecting a sequence of events in the automated response 
system, with reference to a contact-flow file that summarizes 
the contact- flow model of the automated response system; and 

cross- tabulate routing information obtained from 
the detected sequence of events with results from the coded 
and annotated agent-contactor interactions. 
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